Apparatus and method for displaying images

ABSTRACT

Methods and apparatus provide for: acquiring a still image of a scene at a location; acquiring a moving image of at least one object at the scene taken at a different time of day, a different date, and/or a different season; selecting an image frame from among a plurality of image frames of the moving image as a still image frame; generating a three-dimensional (3D) image from the still image in a three-dimensional (3D) object space; superimposing the still image frame from the moving image onto the three-dimensional (3D) image; displaying the three-dimensional (3D) image on a display screen for a user; and generating the moving image superimposed on the three-dimensional (3D) image, starting at the selected image frame, in response to receiving a command from the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. patent application Ser. No.13/804,106, filed Mar, 14, 2013; which is a continuation ofPCT/JP2011/004324, filed Jul. 29, 2011; which claims priority toJP2010-225365, filed Oct. 5, 2010, the entire disclosures of which arehereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus and a method fordisplaying images.

2. Description of the Related Art

With the prevalence of digital still cameras and digital video cameras,there are increased occasions where still images or moving images havingbeen shot are stored in a computer for later viewing, processing, ordisplaying on the screen of a game device or a television system. It isalso popularly done that the shot moving images are uploaded to aposting site on the Internet so as to share them with the other users.

Among the digital cameras are those capable of shooting panoramicimages, which allow the image taking of panoramic images of wide viewangle with perfect ease. Also in wide use are software tools that cangenerate a panoramic image by stitching together a plurality of imagesshot by a digital camera from different shooting directions.

There is a site named “360cities” (http://www.360cities.net) thataccepts the posting of panoramic images shot by users and show them onthe Internet, so that the users around the world can view the panoramicimages posted.

If the panoramic image is a still image, there may be cases where a userwishes to supply moving images to part of objects captured in thepanoramic image, instead of the still image. For example, consider thatthe panoramic image of a night view is shot. Though the still image ofsuch a night view as a panoramic image looks lovely enough for theviewing, a sense of reality and its actual beauty may be further gainedif moving images are used and the user can see how the illumination ofsome buildings goes on and off, for instance. Also, an active feelingcan be added to the panoramic image if the moving images of a movingobject are shot. Further, when an overall scenery is looked at throughthe panoramic image and then an object of interest is viewed in detailby focusing on the object, provision of moving images to such apanoramic image may result in gathering much information.

SUMMARY OF THE INVENTION

The present invention has been made in view of these problems, and apurpose thereof is to provide a technology for effectively displaying amoving image or moving images within an object image.

In order to resolve the above-described problems, an image displayapparatus according to one embodiment of the present invention includes:a detector configured to extract a feature point between an object imageand a frame of part of a moving image to be pasted onto the objectimage, and configured to detect a control point with which the movingimage are associate with a region of the object image; an alignmentprocessing unit configured to adjust alignment of the moving imagerelative to the object image based on the control point; a mappingprocessing unit configured to map the object image and a still imageframe of the moving image the alignment of which adjusted by thealignment processing unit, onto a three-dimensional (3D) object space asa texture; a three-dimensional (3D) image generator configured togenerate a three-dimensional (3D) object image, when thethree-dimensional object image mapped by the mapping processing unit isviewed in a specified line of sight in such a manner so as to regard ashooting location of the object image as a viewpoint position; a displaycontrol unit configured to display the 3D object image on a screen; andan interface configured to receive an instruction from a user concerningthe displayed 3D object image; and a moving image reproduction unitconfigured to reproduce the moving image, when the interface receives aninstruction to reproduce the moving image onto the 3D object image.

Another embodiment of the present invention relates also to an imagedisplay apparatus. The apparatus includes: an alignment processing unitconfigured to adjust alignment of frame of at least part of a movingimage relative to an object image, when upon receipt of the object imageand the moving image to be pasted onto the object image; a displaycontrol unit configured to display both a still image frame of themoving image, the alignment of which adjusted by the alignmentprocessing unit, and the object image in a manner such that the stillimage frame thereof is superimposed onto the object image; an interfaceconfigured to receive an instruction from a user concerning the objectimage displayed by the display control unit; and a moving imagereproduction unit configured to reproduce the moving image, when theinterface receives an instruction to reproduce the moving image onto thethree-dimensional object image. When the interface receives aninstruction to zoom in on a region containing the moving image in theobject image, the moving image reproduction unit starts to reproduce themoving image instead of the still image frame of the moving image.

Still another embodiment of the present invention relates to a methodfor displaying images. The method includes: reading out, from a storagedevice by a processor, an object image and a moving image to be pastedonto the object image, and configured to adjust alignment of a frame ofat least part of the moving image relative to the object image,displaying, by a processor, a still image frame of the moving image, thealignment of which adjusted, and the object image in a manner such thatthe still image frame thereof is superimposed onto the object image;receiving, by a processor, an instruction from a user concerning thedisplayed object image; and initializing reproduction, by a processor,of the moving image instead of the still image frame of the movingimage, when an instruction is received to zoom in on a region containingthe moving image on the object image.

Optional combinations of the aforementioned constituting elements, andimplementations of the invention in the form of methods, apparatuses,systems, computer programs, data structures, recording media and soforth may also be effective as additional modes of the presentinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of examples only, withreference to the accompanying drawings which are meant to be exemplary,not limiting, and wherein like elements are numbered alike in severalFigures in which:

FIG. 1 is a configuration diagram of a panoramic image display apparatusaccording to an embodiment;

FIG. 2 shows a structure of a controller, connected to the panoramicimage display apparatus of FIG. 1, which is an example of an inputdevice;

FIGS. 3A to 3D are illustrations with which to explain the mechanism andshooting directions of an omnidirectional image shooting system used toshoot panoramic images;

FIG. 4A is an illustration with which to explain azimuth angle θ of acamera;

FIG. 4B is an illustration with which to explain elevation angle φ of acamera;

FIGS. 5A to 5C are illustrations with which to explain a panoramic imageshot when an initial position of a camera is in a direction of azimuthangle θ;

FIGS. 6A to 6C are illustrations with which to explain a panoramic imageshot when a camera is in a direction of elevation angle φ=60°;

FIG. 7A explains a method of how a panoramic image is created bystitching a plurality of images together;

FIG. 7B explains a method of how a panoramic image is created bystitching a plurality of images together;

FIG. 8 is a flowchart to explain a procedure for generating a panoramicimage by the panoramic image display apparatus of FIG. 1;

FIG. 9A and FIG. 9B are examples of a panoramic image and a moving imageto be pasted onto the panoramic image, respectively;

FIG. 10 shows another moving image to be pasted onto a panoramic image;

FIG. 11 is a panoramic image to which the moving image of FIG. 10 hasbeen pasted;

FIG. 12 is an image displayed when the panoramic image shown in FIG. 11is zoomed in on a region thereof where a moving image has been pasted;

FIG. 13 shows a panoramic image to which a moving image has been pasted;

FIGS. 14A, 14B and 14C each shows how a moving image is reproduced byzooming in on a region of the moving image in a panoramic image;

FIG. 15 is an example of a screen when an image displayed on the screenhas returned to the panoramic image of FIG. 13 through a zoom-out; and

FIG. 16 is an example obtained when a moving image, the alignment ofwhich has been readjusted, is pasted onto a panoramic image.

DETAILED DESCRIPTION OF THE INVENTION

A description will be given of an outline of a preferred embodiment.Still image frames in moving images are aligned and stitched into apanoramic image. When a region in a moving image or moving images in themoving images is/are zoomed in, the reproduction of the moving imagesstarts in place of the still image frames that have been displayed sofar. Assume herein that the moving images shot contain a camera-shakecomponent. Then an artifact or noise may occur in an image at a boundaryregion of the moving images with the panoramic image. Thus, an adverseeffect of camera shake is reduced by alpha-blending the moving imagesand the panoramic image at the boundary region. As the reproduction ofthe moving images is interrupted and a region is zoomed out, the framedisplayed last is again alighted to the panoramic image. Or the regionmay be displayed by switching to the original panoramic image when it iszoomed out.

FIG. 1 is a configuration diagram of a panoramic image display apparatus100 according to a preferred embodiment. For example, the panoramicimage display apparatus 100 as shown in FIG. 1 may be functionallyconfigured such that hardware, software or a combination of both isimplemented to a personal computer, a game device, a portable device, amobile terminal and so forth. Part of such functional components may beimplemented in a client, so that the panoramic image display apparatus100 may be realized as a server-client system via a network.

A panoramic image/additional data storage 24 stores panoramic imageshaving information on shooting locations and information on shootingorientations associated with each other. The additional data, such asinformation on shooting locations and shooting orientations, may beadded directly to a data file of panoramic images, or may be managed asa separate file from the panoramic images.

The information on shooting locations includes, for instance,information on latitudes and longitudes given by GPS (Global PositioningSystem). The information on shooting orientation includes, for instance,information on the azimuth (angle of orientation) of the center point ofa panoramic image obtained from an azimuth sensor, and may alsoadditionally include information on the elevation angle and roll angleof a camera at the time of shooting.

If the azimuth of the center point of a panoramic image is given asinformation on a shooting orientation, then it is possible to calculatethe orientation of an arbitrary point of the panoramic image based onthe angle of panning the camera to the right or left. The panoramicimages may have, as the information on the shooting orientations, thecoordinate values of pixels in the orientations of true north, truesouth, true east, and true west of the panoramic images that arecalculated based on the azimuths and pan angles of the center points ofthe panoramic images.

A panoramic image acquiring unit 10 acquires a panoramic image to bedisplayed from the panoramic image/additional data storage 24. Thepanoramic image to be displayed is identified as the user specifies ashooting location on a map or the like.

A moving image data storage 26 stores moving images associated with theinformation on shooting locations. The information on shooting locationsincludes, for instance, information on latitudes and longitudes given byGPS at the time of shooting. The information on shooting locations maybe a specific name of an object shot such as a geographical name and thename of facilities.

A moving image acquiring unit 11 acquires moving image(s), which is/areto be pasted onto a panoramic image, from the moving image data storage26. The moving image acquiring unit 11 may acquire the information onshooting locations acquired by the panoramic image acquiring unit 10from the panoramic image/additional data storage 24 and then acquiremoving images shot at the same shooting location or its vicinity fromthe moving image data storage 26. The shooting locations may beidentified using the information on latitudes and longitudes or may beindirectly identified by specifically identifying the name of a placeand the facility name.

If there are a plurality of sets of moving images shot at the samelocation but shot at different times of day or different seasons, themoving image acquiring unit 11 may select moving images suitable to thetimes of day and the season when the panoramic image was shot. If, apanoramic image is shot in the spring, moving images shot in the springwill be selected; if it is shot during night hours, those shot at nightwill be selected.

A control point detector 20 extracts feature points between a panoramicimage and a frame or frames of moving images and thereby detects controlpoints that associate the panoramic image with the moving images. Sinceprocessing is time consuming if the feature points of all frames of themoving images are to be extracted, only several frames from thebeginning of the moving images are selected, for example, and thencontrol points between the panoramic image and those several framesthereof are extracted. A known art used for the image matching may beused as a feature point extraction process.

Upon completion of the processing of the feature point extraction,control points best suited for the association of a frame of the movingimages with a region of the panoramic image is selected from among theextracted feature points. Though a multiplicity of feature points areextracted, it is desirable that the control points be evenly scatteredover the entire region. More preferably, selected are the control pointslocated near or at grid points, which are obtained when the frame of themoving images are divided into a plurality of regions in grid, so thatthe frame of the moving images can be aligned with great accuracy to thepanoramic image.

An alignment processing unit 12 adjusts the alignment of the movingimages relative to the panoramic image, based on the control points.More specifically, the size of the moving images is changed by enlargingor reducing the moving images so that the control points of thepanoramic image can correspond to those of the frame of the movingimages. Since the panoramic image has a distortion caused by awide-angle image taking, the frame of the moving images should bedeformed according to the distortion in the panoramic image. As aresult, a pan angle, a tilt angle, a roll angle, a scale and the like ofthe frame of moving image are required and thus obtained to map theframe of the moving image into the panoramic image.

Note that the control points are set for the several frames from thebeginning of the moving images, for example. Thus, each of those framesis aligned to a region of the panoramic image, based on the controlpoints. Here, the optimum pan angle, tilt angle, roll angle and scalefor each of those frames may be determined by a method of least squares.The moving images are associated with the region of the panoramic image,based on the finally determined alignment. A frame of moving image,which is, for example, the leading frame of moving image, is pasted ontothe region of the panoramic region, as a still image. Here, the singleframe of moving image pasted thereon is a frame obtained, after thealignment processing has been done, based on the finally determined panangle, tilt angle, roll angle and scale.

A mapping processing unit 14 processes a mapping of a panoramic imageand the alignment-processed still image frame of moving images into athree-dimensional panoramic space as textures.

In the case of a spherical, or omnidirectional (celestial), panoramicimage, a sphere is assumed as a three-dimensional panoramic space, andthe panoramic image is texture-mapped onto the spherical surface by asphere mapping. Or a cube may be assumed as a three-dimensionalpanoramic space, and the panoramic image may be texture-mapped onto thecubic surface by a cube mapping. Also, in the case where the panoramicimage does not have any component in tilt directions and spreads only inthe panning directions, a cylinder may be assumed as a three-dimensionalpanoramic space, and the panoramic image may be texture-mapped onto thecylindrical surface by a texture mapping. The same applies to the casewhere the panoramic image does not have any component in the panningdirections and spreads only in tilt directions.

A 3D image generator 16 generates a three-dimensional (3D) panoramicimage when the 3D panoramic space having a panoramic image and a stillimage frame mapped thereon by the mapping processing unit 14 is viewedin a specified line of sight. When the 3D panoramic space is a sphere,the viewpoint is placed at the center of the sphere. When the 3Dpanoramic space is a cube, the viewpoint is placed at the center of theinterior of the cube. And when the 3D panoramic space is a cylinder, theviewpoint is placed on the center axis of the cylinder. The viewpoint isthe location where the panoramic image to be displayed is shot, and theline of sight is the direction in which the surrounding area is viewedand is thus identified by the azimuth and the elevation angle. The 3Dimage generator 16 generates a 3D image when the 3D panoramic space isviewed in the line of sight identified by the azimuth and the elevationangle.

A moving image reproduction unit 22 reproduces the moving images pastedwithin a 3D panoramic image generated by the 3D image generator 16 andthen supplies it to a display control unit 18. A link to the movingimage data stored in the moving image data storage 26 is provided to theregion where the moving images in the panoramic image are pasted. Themoving image reproduction unit 22 reads out the moving image data, whichis coded based on the link information, from the moving image datastorage 26, then decodes the coded moving image data, and reproduces thethus decoded moving image data.

A display control unit 18 has the thus generated 3D panoramic imagedisplayed on a screen of a display unit. Also, the moving imagereproduction unit 22 displays the moving images reproduced by the movingimage reproduction unit 22, on the screen of the display unit.

The reproduction of moving images starts automatically when, forexample, the region of the panoramic image, where a still image frame ofmoving images is pasted, is zoomed in. Also, a symbol indicating thepresence of moving images in the region of the panoramic image where astill image frame of moving images is pasted may be displayed, so thatthe moving images can be reproduced when a user selects the symbol.

Assume that, during reproduction of moving images, the pan angle, thetilt angle and/or the roll angle of a camera is/are varied betweenframes and that the zoom factor is also varied, for example, andtherefore the shooting direction or shooting magnification ratio isgreatly varied. In such a case, the processing of the feature pointextraction may be carried out between several frames after the change inthe shooting direction or shooting magnification ratio (such framesbeing called “latest frames” or “most recent frames” also) and thepanoramic image, and then new control points may be detected from theextracted feature points. As a result, the information regarding thenewly detected control points may be supplied to the alignmentprocessing unit 12.

In this case, the alignment unit 12 readjusts the alignment of themoving images relative to the panoramic image, based on the new controlpoints and associates the most recent frames of the moving images withthe region of the panoramic image. The mapping processing unit 14 mapsthe finally-displayed frames of the moving images that have beensubjected to the alignment processing, together with the panoramicimage, into the 3D panoramic space. As a result, the moving images areappropriately aligned to the panoramic image based on the most recentinformation on alignment, and the moving images aligned thereto aredisplayed within the panoramic image.

A first alignment processing of the moving images is carried out basedon the control points detected from several frames including the framesat the beginning of the moving images and its subsequent ones. As framesduring reproduction based on a first alignment result are displayed inthe region of the panoramic image, a mismatch or inconsistency may occurif the shooting direction or shooting magnification ratio of the camerais varied. In such a case, control points of the frames duringreproduction are detected anew and then the alignment is readjusted, sothat the frames of the moving images during reproduction thereof can beaccurately associated with the region of the panoramic image.

If, in particular, the shooting location moves while the moving imagesare taken, the position of the panoramic image to which the framesduring reproduction are to be pasted will be moved from the pastingposition determined by the first alignment processing. Thus, the framesduring reproduction can be properly associated with the moved positionof the moving images if the control points are detected again and thealignment processing is carried out again.

The alignment of the moving images may be readjusted when thereproduction of moving images end or is interrupted. The moving imagesare reproduced when a region of the panoramic image, to which a stillimage frame or still image frames of the moving images are pasted, iszoomed in. Thus, even though the frames during reproduction are notaligned appropriately, such a misalignment may be negligible. However,if the reproduction of moving images ends or is interrupted and a framedisplayed at the end of the moving images (hereinafter referred to as a“last displayed frame”) after the moving images have been zoomed out andif the last displayed frame is not properly aligned to the panoramicimage, such a misalignment may look unnatural. In the light of this, thecontrol point detector 20 may perform the processing of the featurepoint extraction between the last displayed frame and the panoramicimage so as to newly detect control points, and the alignment processingunit 12 may readjust the alignment of the moving images relative to thepanoramic image, based on the newly detected control points. As aresult, the last displayed frame of the moving images is associated withthe region of the panoramic image. Thereby, the last displayed frame ofthe moving images, namely the last frame if the reproduction of themoving images has been completed or namely a frame displayed at the timeof interruption if the reproduction thereof has been interrupted in themiddle of reproduction, is appropriately aligned to the panoramic imageand displayed within the panoramic image.

If, during reproduction of moving images, the shooting direction orshooting magnification ratio varies between frames in excess of apredetermined threshold, the control point detector 20 will new controlpoints between the most recent frame and the panoramic image. However,at the time when the shooting direction or shooting magnification ratiovaries between frames in excess of the predetermined threshold, thealignment processing unit 12 may not readjust the moving image but storeonly the information on the new controls points in memory and thealignment processing unit 12 may readjust the moving images asnecessary, such as when the panoramic image is zoomed out.

A user interface 40 is a graphical user interface through which the usercan manipulate the graphics displayed on the screen of a display usingan input device. The user interface 40 receives user instructions on themap or 3D panoramic image displayed on the screen from the input devicewhich may be a controller of a game device, a mouse, a keyboard, or thelike. FIG. 2 shows a controller 102 as an example of the input device,whose construction will be discussed in detail later. If there is noentry from the input device for a predetermined period of time, anoperation command may be executed on the map or 3D panoramic imagedisplayed on the screen by a script reproduction unit.

The user interface 40 instructs the panoramic image acquiring unit 10 toacquire a specified panoramic image from the panoramic image/additionaldata storage 24.

The user can input instructions to change the line of sight for viewingthe 3D panoramic space by operating an analog stick 118 or directionkeys 116 of the controller 102, for instance. A line-of-sight settingunit 32 of the user interface 40 gives a line of sight instructed by theuser to the 3D image generator 16. The 3D image generator 16 generatesan image when the 3D panoramic space is viewed in a specified line ofsight.

An angle-of-view setting unit 31 sets an angle of view when the user hasperformed a zoom operation on the panoramic image being displayed andgives the information of the angle of view thus set to the panoramicimage acquiring unit 10 and the 3D image generator 16. Where panoramicimages of different angles of view are stored in the panoramicimage/additional data storage 24, the panoramic image acquiring unit 10reads out a panoramic image of an angle of view closest to the set angleof view and changes the panoramic image to be displayed. The 3D imagegenerator 16 realizes the visual effects of zoom-in and zoom-out byenlarging or reducing the 3D panoramic image according to the set angleof view.

A panoramic image may have information on the shooting altitude, and thepanoramic image/additional data storage 24 may store panoramic imagesshot at different altitudes at the same shooting location. In such acase, the user can input instructions to change the altitude byoperating L1/L2 buttons 161 and 162 provided on the left front of thecasing of the controller 102, for instance. Pressing the L1 button 161will give an instruction to raise the altitude, and pressing theL2button 162 will give an instruction to lower the altitude.

The display control unit 18 may indicate to the user, for instance, withsmall arrows at the top and bottom portions of the screen that thepanoramic image currently being displayed has panoramic images shot atdifferent altitudes at the same shooting location. An arrow facingupward in the top portion of the screen indicates the presence of apanoramic image shot at a higher altitude than the current one, and anarrow facing downward in the bottom portion of the screen indicates thepresence of a panoramic image shot at a lower altitude than the currentone.

Upon receipt of an instruction from the user to change the altitude, thealtitude setting unit 34 of the user interface 40 instructs thepanoramic image acquiring unit 10 to acquire a panoramic imagecorresponding to the specified altitude, despite the same shootingplace, from the panoramic image/additional data storage 24. Thepanoramic image acquiring unit 10 acquires a panoramic image of a highershooting altitude than the panoramic image currently being displayedwhen the L1 button 161 is pressed, and acquires a panoramic image of alower shooting altitude than the current one when the L2 button 162 ispressed.

When a display is produced by switching to a panoramic image of adifferent shooting altitude, the display control unit 18 may give aspecial effect to the image so that the user may have a sense of ridingan elevator up or down. For example, when switching to a panoramic imageof a higher altitude, the panoramic image currently being displayed canbe scrolled downward, thereby having the panoramic image of a higheraltitude descend from above with the result that the user may have asense of having risen upstairs.

A panoramic image contains information on the shooting date and time,and the panoramic image/additional data storage 24 may store panoramicimages shot at different dates and times at the same shooting location.In such a case, the user can input instructions to change the date andtime by operating R1/R2 buttons 151 and 152 provided on the right frontof the casing of the controller 102, for instance. Pressing the R1button 151 will give an instruction to shift to a later date and time,and pressing the R2 button 152 will give an instruction to shift to anearlier date and time.

The display control unit 18 may indicate to the user, for instance, withwatch and calendar icons in the corner of the screen that the panoramicimage currently being displayed has panoramic images shot at differentdates and times. Watch icons may be displayed to indicate the presenceof panoramic images for different times of day such as morning, noon,and night, whereas calendar icons may be displayed to indicate thepresence of panoramic images for different seasons such as spring,summer, autumn, and winter.

Upon receipt of an instruction from the user to change the date andtime, the date/time setting unit 36 of the user interface 40 instructsthe panoramic image acquiring unit to acquire a panoramic imagecorresponding to a specified date and time at the same shooting locationfrom the panoramic image/additional data storage 24. The panoramic imageacquiring unit 10 acquires a panoramic image of a later shooting dateand time than the panoramic image currently being displayed when the R1button 151 is pressed, and acquires a panoramic image of an earliershooting date and time than the current one when the R2 button 152 ispressed.

Thus, it is possible to switch the panoramic image being displayed topanoramic images of a different time of day or season at the sameshooting location, for example, from one shot in the morning to one shotat night, or from one shot in the spring to one shot in the autumn. Inchanging the panoramic image, the display control unit 18 may give aneffect of fade-in and fade-out to the image.

A viewpoint position setting unit 30 set the shooting location of apanoramic image as a viewpoint position and conveys it to the 3D imagegenerator 16. The line-of-sight setting unit 32 sends the specifiedline-of-sight to the 3D image generator 16.

FIG. 2 shows a structure of a controller, connected to the panoramicimage display apparatus of FIG. 1, which is an example of an inputdevice. The panoramic image display apparatus 100 may be a game device,for instance.

The controller 102 has a plurality of buttons and keys to receivecontrol inputs to the panoramic image display apparatus 100. As the useroperates on the buttons or keys of the controller 102, their operationinputs are transmitted to the panoramic image display apparatus 10through wireless or wired connections.

Provided on a casing top surface 122 of the controller 102 are a groupof arrow keys 116, analog sticks 118, and a group of operation buttons120. The group of direction keys 116 include “up-”, “down-”, “left-”,and “right-” direction indication keys. The group of operation buttons120 include a circle button 124, a cross button 126, a square button128, and a triangle button 130.

The user holds a left-hand grip 134 b with the left hand and holds aright-hand grip 134 a with the right hand, and operates the group ofdirection keys 116, the analog sticks 118, and the group of operationbuttons 120 on the casing top surface 122.

Provided on a front side of the controller 102 are a right-handoperation part 150 and a left-hand operation part 160. The right-handoperation part 150 includes an R1 button and an R2 button, whereas theleft-hand operation part 160 includes an L1 button 161 and an L2 button162.

The user can shift a pointer displayed on the screen in vertical andhorizontal directions by operating the directional key group 116. Forexample, when selecting one of a plurality of markers displayed within apanoramic image, the user can shift the pointer between the plurality ofmarkers on the screen by operating the directional key group 116. Theuser can select a desired marker by pressing the circle button 124 whenthe pointer has come upon the marker.

Different functions may be assigned to the respective buttons ofoperation buttons 120 by a panoramic image display application program.For example, the function to specify the display of a menu is assignedto the triangle button 130, the function to specify the cancel of aselected item is assigned to the cross button 126, the function tospecify the determination of a selected item is assigned to the circlebutton, and the function to specify the display/non-display of table ofcontents or the like is assigned to the square button 128.

The analog sticks 118 have means to output analog values as they aretipped by the user. The controller 102 sends an analog output signalcorresponding to the direction and amount of tipping of the analog stick118 to the panoramic image display apparatus 100. For example, the usercan shift the viewpoint in a desired direction within a 3D panoramicimage shown on the display by tipping the analog stick 118 in thedesired direction.

The casing top surface 122 is further provided with an LED button 136, aselector button 140, and a start button 138. The LED button 136 is usedas the button for the display of the menu screen on the display, forinstance. The start button 138 is the button with which the userinstructs the start of a panoramic image display application, the startor pause of playback of a panoramic image, or the like. The selectorbutton 140 is the button with which the user instructs a selection froma menu display shown on the display or the like.

FIGS. 3A to 3D are illustrations with which to explain the mechanism andshooting directions of an omnidirectional image shooting system 230 usedto shoot panoramic images.

As shown in FIG. 3D, a camera 200 in the omnidirectional image shootingsystem 230 is secured onto a control disk 210. And a camera's pan anglecan be changed as the control disk 210 is rotated around a Z axis, acamera's tilt angle can be changed as the control disk 210 is rotatedaround an X axis, and a camera's roll angle can be changed as thecontrol disk 210 is rotated around a Y axis. The Z axis herein is thevertical axis (gravitational direction axis).

FIG. 3A is a top view of the camera 200 installed on the control disk210. The initial position (Y-axis direction) of the control disk is panangle 0°, and the pan angle can be changed within a range of −180° to+180° around the Z axis.

FIG. 3B is a front view of the camera 200 installed on the control disk210. The horizontal state of the control disk 210 is roll angle 0°, andthe roll angle can be changed within a range of −180° to +180° aroundthe Y axis.

FIG. 3C is a side view of the camera 200 installed on the control disk210. The horizontal state of the control disk 210 is tilt angle 0°, andthe tilt angle can be changed within a range of −90° to +90° around theX axis.

In order to endow a panoramic image shot by the omnidirectional imageshooting system 230 of FIG. 3D with information on the shootingorientations, it is necessary to record the orientations of the camera200 at the time of image taking. For that purpose, the omnidirectionalimage shooting system 230 is provided with an azimuth sensor formeasuring orientations and an acceleration sensor for measuring tiltangles. The ominidirectional image shooting system 230 is furtherprovided with a GPS sensor or the like for measuring the shootinglocation and time.

FIG. 4A is an illustration with which to explain azimuth angle θ of thecamera 200, and FIG. 4B is an illustration with which to explainelevation angle φ of the camera 200. FIG. 4A is a top view of the camera200, in which the camera 200 in an initial position of shooting faces adirection 220 which is azimuth angle θ displaced from true north toeast. This direction is equal to pan angle 0°. In other words, theazimuth angle of the reference direction 220 of the pan angle is θ. Whenshooting a panoramic image, the image of an object is takenpanoramically by changing the pan angle in a range of −180° to +180°with respect to the reference direction 220 of the azimuth angle θ.

FIG. 4B is a side view of the camera 200. The elevation angle φ is thedirection of tilt 0°, which is an angle where an upper direction isdefined to be positive in relation to the Y-axis direction, when thecamera 200 is rotated around the X axis. Normally, the elevation angleis 0° since the image taking is done with the camera 200 set in ahorizontal position. To shoot a spherical panoramic image, however, itis necessary to take the images of the object by changing the elevationangle φ with the tilt of the camera.

FIGS. 5A to 5C are illustrations with which to explain a panoramic imageshot when the initial position of the camera 200 is in a direction ofthe azimuth angle θ.

As shown in the top view of FIG. 5A, the camera 200 in the initialposition faces the direction 220 of azimuth angle θ. And as shown in theside view of FIG. 5B, the elevation angle of the camera 200 is φ=0°.With the elevation angle kept at φ=0°, an omnidirectional panoramic viewis shot at the elevation angle φ=0° while the pan angle of the camera200 with respect to the reference direction 220 is varied within a rangeof −180° to +180°. FIG. 5C is a panoramic image 300 taken in theabove-described manner. At the center of the panoramic image 300, thepan angle is 0°. The left half of the panoramic image 300 is an imagesuch that it is taken by varying the pan angle within a range of 0° to−180°. Similarly, the right half of the panoramic image 300 is an imagesuch that it is taken by varying the pan angle within a range of 0° to180°.

The central position of the pan angle 0° is displaced from true north byazimuth angle θ. Thus, the positions of north (N), south (S), east (E),and west (W) are those indicated by dotted lines. As long as thepanoramic image 300 contains the azimuth angle θ of the central positionof pan angle 0° as the information on the shooting orientations, thepixel positions of north (N), south (S), east (E), and west (W) can beevaluated in consideration of a displacement of the azimuth angle θ.Alternatively, instead of the azimuth angle θ, the coordinate values ofpixel positions of north (N), south (S), east (E), and west (W) may beused as the information on the shooting orientations.

In order to obtain a spherical panoramic image, it is necessary to takeimages by varying the elevation angle of the camera 200. For example, ifthe angle of view of the camera 200 is 60°, a spherical panoramic imagecan be theoretically obtained as follows. That is, the camera 200 istilted vertically at ±60°, and the similar image taking is done byvarying the pan angle within a range of −180° to +180°.

FIGS. 6A to 6C are illustrations with which to explain a panoramic imageshot when a camera 200 is in a direction of elevation angle φ=60°. Asshown in the top view of FIG. 6A, the camera 200 in the initial positionfaces the direction 220 of azimuth angle θ. And as shown in the sideview of FIG. 6B, the elevation angle of the camera 200 is φ=0°. With theelevation angle kept at φ=60°, a panoramic view 302 as shown in FIG. 6Cis shot at the elevation angle φ=60° while the pan angle of the camera220 with respect to the reference direction 220 is varied within a rangeof −180° to +180°.

With the elevation angle kept at φ=−60°, a panoramic view 302 issimilarly shot at the elevation angle φ=−60° while the pan angle isvaried within a range of −180° to +180°. A spherical panoramic image isobtained by combining the panoramic images shot at the elevation anglesφ=0°, 60°, and −60. However, in implementation, a method is oftenemployed where the vicinities of a boundary (bordering areas) are takenin an overlapped manner, in order to correct the mismatch caused by lensdistortions when images are stitched together in boundary portions atthe angle of view.

The spherical panoramic image obtained as described above is endowedwith information on azimuth angles and elevation angles. Therefore, itis possible to identify the azimuth and elevation angle of an arbitrarypixel of the panoramic image based on the information. Also, thepanoramic image is provided with the latitude and longitude informationmeasured by GPS as the positional information of the shooting location.The additional information to be attached to the panoramic image may berecorded, for example, in the format of image file called Exif(Exchangeable Image File Format). The place-name of the shootinglocation can be recorded in a part of the file name, whereas theshooting date and time, the latitude and longitude of the shootinglocation, the altitude, the azimuth angle, and the like can be recordedas data in the Exif format. The elevation angle, which is not defined inthe Exif format, is recorded as extended data.

FIG. 7A and FIG. 7B are illustrations with which to explain a method ofcreating a panoramic image by stitching a plurality of images together.

In the example of FIG. 7A, seven images 341 to 347 shot by tilting (orpanning) the camera 200 are mapped into a cylinder and then stitchedtogether to prepare a cylindrical image 340. When the images arestitched together, the bordering areas of the images are overlapped witheach other.

As shown in FIG. 7B, a plurality of cylindrical images like one shown inFIG. 7A are obtained in the panning (or tilting) direction by theshooting with the panning (or tilting) of the camera 200. Anomnidirectional panoramic image 360 is finally obtained by synthesizingthese cylindrical images 340 a to 340 f with the bordering areas of theimages overlapped.

FIG. 8 is a flowchart to explain a procedure for generating a panoramicimage by the panoramic image display apparatus 100. Each step in theprocedure, for generating the panoramic image, shown in FIG. 8 byreferring to FIGS. 9A and 9B to FIG. 16. In the flowchart shown in FIG.8, the procedure of each structural component is shown using S (thecapital letter of “Step”), which means a step, and numbers combined. Ifa determining process is executed in a processing indicated by thecombination of S and a number and if the decision result is positive,“Y” (the capital letter of “Yes”) will be appended like “(Y of S24)”.If, on the other hand, the decision result is negative, “N” (the capitalletter of “No”) will be appended like “(N of S24)”.

The panoramic image acquiring unit 10 acquires panoramic images 400 shotand information on the shooting locations appended to the panoramicimages 400, from the panoramic image/additional data storage 24 (S10).The moving image acquiring unit 11 acquires moving images shot at thesame shooting location or near their shooting location, from the imagedata storage 26 (S12).

The control point detector 20 acquires several frames of the movingimages 500 starting from the beginning thereof (S14), extracts featurepoints by which the matching may be obtained between the panoramic image400 and the frames of the moving images 500, and selects control pointsby which to position the frames of the moving image 500 in a region ofthe panoramic image 400 (S16).

FIG. 9A and FIG. 9B are examples of a panoramic image 400 and a movingimage 500 to be pasted onto the panoramic image 400, respectively. Inthis case, four control points are detected between the panoramic imageof FIG. 9A and a leading frame of the moving images 500, and therespective control points associated with the two images are associatedwith each other using the dotted lines.

The alignment processing unit 12 adjusts the alignment of the movingimage 500 relative to the panoramic image 400, based on the controlpoints, and associates the moving image with the region of the panoramicimage 400 (S18).

FIG. 10 shows another moving image 510 to be pasted onto a panoramicimage. The moving images 510 are those taken of a Ferris wheel. FIG. 11is a panoramic image 410 to which the moving image 510 of FIG. 10 hasbeen pasted. The panoramic image 410 has the image of the Ferris shot inthe first place, and a still image of the leading frame of the movingimages 510 for the Ferris wheel is pasted onto the region where theFerris while was shot. In another exemplary embodiment, though themoving images of the Ferris wheel are not taken on the panoramic image410, the moving image 510 may be pasted onto the panoramic image 410, inwhich case the Ferris wheel can be virtually located in a position ofthe panoramic image 410.

Referring back to the flowchart of FIG. 8, the mapping processing unit14 maps the panoramic image 400 associated with the moving images 500and 510, onto a panoramic sphere (S20). The 3D image generator 16generates a three-dimensional (3D) panoramic image when the 3D panoramicsphere, onto which the moving images 500 and 510 have been mapped, isviewed in a specified line of sight and then the display control unit 18displays the thus generated 3D panoramic image (S2).

If the user zooms in on regions including the moving images 500 and 510pasted onto the panoramic images 400 and 410, respectively (Y of S24),the moving image reproduction unit 22 will start to reproduce the movingimages 500 and 510. Thereby, the scenes showing how the illumination ofsome buildings goes on and off and how an elevator goes up and down arereproduced in the moving images 500, and the scenes of how the Ferriswheel rotates is reproduced in the moving images 510. If no zoom-inoperation is performed on the moving images 500 and 510 (N of S24), theprocess of generating the panoramic images is terminated.

When generating the moving images, the moving image reproduction unit 22alpha-blends the boundary region of the moving images 500 and 510 withthe panoramic images 400 and 410 (S26).

FIG. 12 is an image 412 displayed when the panoramic image 410 shown inFIG. 11 is zoomed in on the region thereof where the moving images 510have been pasted. Reproduction of the moving images 510 results in themovement of the boundary for each frame in a boundary region 512 of themoving images 510 with the panoramic image 410 because of acamera-shake. Therefore, blur or mismatch may occur in the boundaryregion. This is because several frames from the beginning of the movingimages 510 are aligned to the panoramic image but the control points ofall frames of the moving images do not necessarily match the controlpoints of the panoramic image in consequence of the camera-shake. Theblur or mismatch in the boundary region resulting from the camera-shakedoes not occur when the still image frames of the moving images 510 arepasted onto the panoramic image 410 as in FIG. 11 but it occurs when theregion is zoomed in and then the moving images 510 are reproduced.

Thus, the moving image reproduction generator 22 alpha-blends the pixelsof each frame of the moving images 510 and pixels of the panoramic image410 at the boundary region 512 of the moving images 510 with thepanoramic image 410 so as to carry out a post-processing. This can makethe blur or mismatch less conspicuous.

If, during reproduction of moving images, the variation in the shootingdirection or shooting magnification ratio between frames is within apredetermined threshold value (N of S28), the process of Step S26 willbe repeated. If the shooting direction or shooting magnification ratiobetween frames varies in excess of the predetermined threshold value (Yof S28), the moving image reproduction unit 22 will acquire severalframes after the change in the shooting direction or shootingmagnification ratio (“latest frames” or “most recent frames”) (S30). Andthe procedure returns to Step S16, the control point detector 20 detectsnew control points with which to associate the most recent frames of themoving images with the region of the panoramic image, and the alignmentprocessing unit 12 readjusts the alignment of the moving images based onthe newly detected control points so as to relate the last displayedframe of the moving images to the panoramic image.

With reference to FIG. 13 to FIG. 16, a description is given of thereadjustment of the alignment of moving images. FIG. 13 shows apanoramic image 420 to which moving images 520 have been pasted. Themoving images 520 are those taken of a horse and a carriage. Thematching is obtained between the background image of the moving images520 and the moving images 520 themselves and thereby the leading frameof the moving images 520 is associated with a region of the panoramicimage 420 and then pasted onto the region thereof.

FIGS. 14A, 14B and 14C each shows how the moving images 520 arereproduced by zooming in on a region of the moving images 520 in thepanoramic image 420. Frames in FIG. 14A, FIG. 14B, and FIG. 14C arethose reproduced in order of time (left to right). The carriage advancesfrom right to left in the panoramic image 420 and reach in front of theFerris wheel.

FIG. 15 is an example of a screen when an image displayed on the screenhas returned to the panoramic image of FIG. 13 through a zoom-out. Whenthe image is zoomed out, the reproduction of the moving images isinterrupted and the last displayed frame is displayed. Or thereproduction of the moving images may continue even after the zoom-outand then the most recent frames may be displayed. For comparison, adescription is given herein of a case where the alignment is notreadjusted. The moving images are aligned at the right side of thepanoramic image 420 based on the first alignment result of the movingimages 520 but the image being displayed is the last displayed frame orthe most recent frame of FIG. 14C. Thus the background of the movingimages 520, which does not match the panoramic image 420, looksunnatural.

FIG. 16 is an example obtained when the moving images 520, the alignmentof which has been readjusted, are pasted onto the panoramic image 420.The control points of the background image in the last displayed frameof the moving images 520 are associated with the control points of adestination region in the panoramic image 420, so that the background ofthe moving images 520 matches the panoramic image 420. If, as describedherein, an object (the carriage in this example) shot in the movingimages 520 moves from one spot to another, the position at which themoving images 520 are pasted onto the panoramic image 420 will be moved,too, following the object.

If the reproduction of moving images is interrupted after a zoom-out andif the leading frame, instead of the last displayed frame, is to bedisplayed, the moving images 520 will preferably be aligned, as shown inFIG. 15, at the right side of the panoramic image 420 based on the firstalignment result thereof.

The position at which the moving images 520 are pasted may be moved atthe time of a zoom-out. Also, the position at which the moving images520 are pasted may be moved to within the panoramic image 420 in realtime by adjusting the control points of each frame of the moving images520 to the control points in the panoramic image 420. While an image isbeing zoomed in, a part of the panoramic image 420 may be displayedaround the moving images 520. In such a case, the pasting position ofeach frame is moved to within the panoramic image during reproduction ofthe moving images 520, so that the background image of each frame can bemade to match the panoramic image 420 and therefore the continuity canbe maintained.

As described above, the panoramic image display apparatus according tothe present embodiments can accurately paste the still image frame orframes of the moving images to the panoramic image, based on theacquired control points of several frames from the beginning of themoving images and can display the panoramic image with the still imageframe or frames accurately pasted thereonto. The reproduction of movingimages starts automatically when the region of the panoramic image,where moving images are pasted, is zoomed in. Though the positions ofcontrol points differ frame by frame due to a camera-shake, artifact orblur at the boundary region can be made less conspicuous byalpha-blending the pixel values of the moving images and the panoramicimage at the boundary region of the moving images with the panoramicimage.

As the reproduction of moving images is interrupted, the control pointsof a frame or frames displayed at the time of interruption is/areacquired and then the frame or frames of the moving images are pastedonto the panoramic image. Thus the continuity with the panoramic imagecan be maintained when the frame or frames are zoomed out. Ifparticularly the object moves and the shooting direction varies, theposition at which the moving images are to be pasted is moved within thepanoramic image, so that the continuity can be maintained between themoving images and the panoramic image.

The present invention has been described based upon illustrativeembodiments. These embodiments are intended to be illustrative only andit will be obvious to those skilled in the art that variousmodifications to the combination of constituting elements and processescould be developed and that such modifications are also within the scopeof the present invention.

In the foregoing description, a panoramic image associated with movingimages is mapped into the 3D panoramic space such as a sphere and then a3D panoramic image when the 3D panoramic space is viewed in a specifiedline of sight is displayed on the screen. However, the panoramic imageassociated with the moving images may be simply displayedtwo-dimensionally. In this case, there is no need for construction ofthe mapping processing unit 14 and the 3D image generator 16, thussimplifying the panoramic image display apparatus 100. In this case,too, as the region of the moving images associated with thetwo-dimensionally displayed panoramic image is zoomed in, the movingimages are automatically reproduced. Or the reproduction of the movingimages may be started when the user selects a symbol indicating that themoving images are associated with the panoramic image.

Panoramic images herein are not limited to ones shot by theomnidirectional image shooting system as shown in FIG. 3, but they maybe ones shot through a fish-eye lens or ones synthesized or merged froma plurality of images shot with a normal digital camera in differentshooting directions.

In the foregoing description, the alignment processing unit 12associates the moving images with a region of the panoramic image andthen the mapping processing unit 14 maps the panoramic image and thestill image frames of the moving images into a 3D panoramic space. Asanother method, the alignment processing unit 12 may synthesize or mergethe still image frames of the moving images into the panoramic image,besides the alignment processing, based on the control points and thenmay map a panoramic image into which the still image frames of themoving images have been combined, into the 3D panoramic space.

Of the functional components of the panoramic image display apparatus100, the components related to a function for mainly aligning andlinking the moving images to the panoramic image may be implemented to aserver, while the components related to a function for mainly viewing apanoramic image and the moving images linked to the panoramic image maybe implemented to a client. Thereby, the panoramic image displayapparatus 100 can be realized as a server-client system via a network.The server may carry out a process of aligning and linking the stillimage frames of the moving images to the panoramic image, and aninterface with which to view the panoramic view linked to the movingimages may be provided to the client. As the user zooms in on a regionof the moving images on a display screen of the client, content of thecoded streams of the moving images may be delivered from the server andthen the client may decode the content and reproduce the moving images.

The “panoramic image” as used in this patent specification is notlimited to a “panorama” image in the narrow sense, namely a horizontallyor vertically long image or a panoramic view of 360 degrees, but simplyan image that covers a wide range of area. Also, explained in theembodiments is an example where the moving images are pasted onto thepanoramic image, which is an object image. Note that the object imagedoes not have to be a so-called panoramic image and that the presentinvention is applicable to a normal image of arbitrary size serving asthe object image. Or the object image may be an image where a pluralityof images having different resolutions are hierarchized. Such ahierarchized image may be constructed such that when a partial region ofimage is enlarged, the enlarged region is replaced by an image of ahigher resolution.

What is claimed is:
 1. A method, comprising: acquiring a still image ofa scene at a location; acquiring a moving image of at least one objectat the scene taken at a different time of day, a different date, and/ora different season; selecting an image frame from among a plurality ofimage frames of the moving image as a still image frame; generating athree-dimensional (3D) image from the still image in a three-dimensional(3D) object space; superimposing the still image frame from the movingimage onto the three-dimensional (3D) image; displaying thethree-dimensional (3D) image on a display screen for a user; andgenerating the moving image superimposed on the three-dimensional (3D)image, starting at the selected image frame, in response to receiving acommand from the user.
 2. The method of claim 1, wherein the step ofgenerating the moving image superimposed on the three-dimensional (3D)image includes putting the at least one object into motion from a firstarea on the three-dimensional (3D) image to a second area on thethree-dimensional (3D) image.
 3. The method of claim 1, wherein thecommand from the user is a command to zoom in toward the still imageframe within the displayed three-dimensional (3D) image.
 4. The methodof claim 1, wherein the step of acquiring the still image of the sceneincludes downloading the still image of the scene from a server over adata network.
 5. The method of claim 1, wherein the step of acquiringthe moving image of the scene includes downloading the moving image ofthe scene from a server over a data network.
 6. The method of claim 1,wherein the still image is a panoramic image and the three-dimensional(3D) object space is a three-dimensional (3D) panoramic space.
 7. Themethod of claim 6, wherein the three-dimensional (3D) panoramic space isone of: a spherical three-dimensional (3D) space, a cubicthree-dimensional (3D) space, and a cylindrical three-dimensional (3D)space.
 8. An image display apparatus, comprising: a first acquisitionunit configured to acquiring a still image of a scene at a location; asecond acquisition unit configured to acquire a moving image of at leastone object at the scene taken at a different time of day, a differentdate, and/or a different season; a control unit configured to select animage frame from among a plurality of image frames of the moving imageas a still image frame; an image generating unit configured to generatea three-dimensional (3D) image from the still image in athree-dimensional (3D) object space; a mapping unit configured tosuperimpose the still image frame from the moving image onto thethree-dimensional (3D) image; a display controller unit configured todisplay the three-dimensional (3D) image on a display screen for a user;and generating the moving image superimposed on the three-dimensional(3D) image, starting at the selected image frame, in response toreceiving a command from the user.
 9. The image display apparatus ofclaim 8, wherein the image generating unit generates the moving imagesuperimposed on the three-dimensional (3D) image by putting the at leastone object into motion from a first area on the three-dimensional (3D)image to a second area on the three-dimensional (3D) image.
 10. Theimage display apparatus of claim 8, wherein the command from the user isa command to zoom in toward the still image frame within the displayedthree-dimensional (3D) image.
 11. The image display apparatus of claim8, wherein the first acquisition unit operates to acquire the stillimage of the scene by downloading the still image of the scene from aserver over a data network.
 12. The image display apparatus of claim 8,wherein the second acquisition unit operates to acquire the moving imageof the scene by downloading the moving image of the scene from a serverover a data network.
 13. The image display apparatus of claim 8, whereinthe still image is a panoramic image and the three-dimensional (3D)object space is a three-dimensional (3D) panoramic space.
 14. The imagedisplay apparatus of claim 13, wherein the three-dimensional (3D)panoramic space is one of: a spherical three-dimensional (3D) space, acubic three-dimensional (3D) space, and a cylindrical three-dimensional(3D) space.
 15. An image display apparatus, comprising: a serveroperatively connected to a communications network and configured totransmit at least one of a still image of a scene at a location and amoving image of at least one object at the scene taken at a differenttime of day, a different date, and/or a different season, to a clientterminal connected to the communications network, wherein the clientterminal, upon acquiring the still image of the scene and the movingimage of the scene, operates to: select an image frame from among aplurality of image frames of the moving image as a still image frame;generate a three-dimensional (3D) image from the still image in athree-dimensional (3D) object space; superimpose the still image framefrom the moving image onto the three-dimensional (3D) image; display thethree-dimensional (3D) image on a display screen for a user; andgenerate the moving image superimposed on the three-dimensional (3D)image, starting at the selected image frame, in response to receiving acommand from the user.
 16. The image display apparatus of claim 15,wherein generating the moving image superimposed on thethree-dimensional (3D) image includes putting the at least one objectinto motion from a first area on the three-dimensional (3D) image to asecond area on the three-dimensional (3D) image.
 17. The image displayapparatus of claim 15, wherein the command from the user is a command tozoom in toward the still image frame within the displayedthree-dimensional (3D) image.
 18. The image display apparatus of claim15, wherein the still image is a panoramic image and thethree-dimensional (3D) object space is a three-dimensional (3D)panoramic space.
 19. The image display apparatus of claim 15, whereinthe three-dimensional (3D) panoramic space is one of: a sphericalthree-dimensional (3D) space, a cubic three-dimensional (3D) space, anda cylindrical three-dimensional (3D) space.
 20. A non-transitory,computer-readable storage medium containing a computer program, which isexecutable by a computer in order to cause the computer to carry outactions, comprising: acquiring a still image of a scene at a location;acquiring a moving image of at least one object at the scene taken at adifferent time of day, a different date, and/or a different season;selecting an image frame from among a plurality of image frames of themoving image as a still image frame; generating a three-dimensional (3D)image from the still image in a three-dimensional (3D) object space;superimposing the still image frame from the moving image onto thethree-dimensional (3D) image; displaying the three-dimensional (3D)image on a display screen for a user; and generating the moving imagesuperimposed on the three-dimensional (3D) image, starting at theselected image frame, in response to receiving a command from the user.